perf: optimize spark_cast_float64_to_utf8 (~40% faster)#4918
Conversation
andygrove
changed the title
spark_cast_float64_to_utf8 (~40% faster)
mbutrovich
left a comment
mbutrovich
left a comment
There was a problem hiding this comment.
First pass, thanks @andygrove!
| // is allocated per row. | ||
| let mut builder = GenericStringBuilder::<OffsetSize>::with_capacity( | ||
| array.len(), | ||
| array.len() * 8, |
There was a problem hiding this comment.
GenericStringBuilder::<OffsetSize>::with_capacity(array.len(), array.len() * 8). Eight bytes
per value under-provisions typical fractional output like 1234.5678 (9 bytes) or the
scientific forms this cast produces (4.9E-324 is 8, -1.4E-45 is 8, longer with more
mantissa digits), so the value buffer will still reallocate on realistic data. arrow-rs itself
uses an AVERAGE_STRING_LENGTH of 16 for its own capacity helper
(arrow-array/src/builder/generic_bytes_builder.rs:376).
Suggested change: raise the hint to array.len() * 16 to match arrow-rs's own average and
avoid mid-loop growth on ordinary float text. The cost of a slightly large preallocation is
negligible per that same arrow-rs doc note.
2 participants
Which issue does this PR close?
N/A
Rationale for this change
Optimize existing expression.
What changes are included in this PR?
Format float-to-string casts straight into a GenericStringBuilder with a reused scratch buffer instead of allocating a String per row (and a second one in the scientific-notation path), removing one malloc/free and one copy per value.
How are these changes tested?
Existing tests.
Benchmark (criterion):
Full criterion output: